{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 如何开发单词级神经语言模型并使用它来生成文本"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一个语言模型可以预测序列中的下一个单词的概率。\n",
    "\n",
    "神经网络模型是用于开发统计语言模型的首选方法，因为它们可以使用分布式表示，其中具有相似含义的不同单词具有相似的表示，并且它们在进行预测时可以使用最近观察到的单词的上下文。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 柏拉图的共和国\n",
    "共和国是希腊古典哲学家柏拉图最著名的作品。\n",
    "\n",
    "它的结构是一个对话（例如对话），涉及城市国家内的秩序与正义。\n",
    "\n",
    "整个文本可在公共领域免费获得。可在Gutenberg项目网站上以多种格式获得。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "文本应以如下内容开头：\n",
    "\n",
    "BOOK I.\n",
    "\n",
    "I went down yesterday to the Piraeus with Glaucon the son of Ariston,\n",
    "…\n",
    "\n",
    "以如下内容结尾：\n",
    "\n",
    "…\n",
    "And it shall be well with us both in this life and in the pilgrimage of a thousand years which we have been describing.\n",
    "\n",
    "将清理后的版本另存为“ republic_clean”。txt”在您当前的工作目录中。该文件应为约15802行文本。\n",
    "\n",
    "现在，我们可以根据该文本开发语言模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 语言模型设计\n",
    "在本教程中，我们将开发一个文本模型，然后将其用于生成新的文本序列。\n",
    "\n",
    "语言模型将是统计性的，并将在给定文本输入序列的情况下预测每个单词的概率。预测的单词将作为输入输入，进而生成下一个单词。\n",
    "\n",
    "一个关键的设计决定是输入序列应该有多长。它们需要足够长，以使模型能够学习用于预测单词的上下文。当我们使用模型时，此输入长度还将定义用于生成新序列的种子文本的长度。\n",
    "\n",
    "没有正确的答案。如果有足够的时间和资源，我们可以探索模型学习不同大小的输入序列的能力。\n",
    "\n",
    "取而代之的是，我们将随意选择50个单词的长度作为输入序列的长度。\n",
    "\n",
    "我们可以处理数据，以便该模型只处理独立的句子，并填充或截断文本以满足每个输入序列的这一要求。您可以将其作为本教程的扩展进行探索。\n",
    "\n",
    "相反，为了使示例简短，我们将让所有文本一起流动并训练模型以预测文本中句子，段落甚至书籍或章节中的下一个单词。\n",
    "\n",
    "现在我们有了模型设计，我们可以看一下如何将原始文本转换为由50个输入单词到1个输出单词的序列，准备适合模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 载入文字\n",
    "第一步是将文本加载到内存中。\n",
    "\n",
    "我们可以开发一个小函数来将整个文本文件加载到内存中并返回它。该函数称为load_doc（），并在下面列出。给定文件名，它将返回一系列已加载的文本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load doc into memory\n",
    "def load_doc(filename):\n",
    "\t# open the file as read only\n",
    "\tfile = open(filename, 'r')\n",
    "\t# read all text\n",
    "\ttext = file.read()\n",
    "\t# close the file\n",
    "\tfile.close()\n",
    "\treturn text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BOOK I.\n",
      "\n",
      "I went down yesterday to the Piraeus with Glaucon the son of Ariston,\n",
      "that I might offer up my prayers to the goddess (Bendis, the Thracian\n",
      "Artemis.); and also because I wanted to see in what\n"
     ]
    }
   ],
   "source": [
    "# load document\n",
    "in_filename = './input/republic_clean.txt'\n",
    "doc = load_doc(in_filename)\n",
    "print(doc[:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预处理\n",
    "\n",
    "我们需要将原始文本转换为标记或单词序列，以用作训练模型的源。\n",
    "\n",
    "在查看原始文本的基础上，以下是我们将执行的一些清理文本的特定操作。作为扩展，您可能想自己探索更多操作。\n",
    "\n",
    "- 用空格替换“ –”，以便我们更好地拆分单词。\n",
    "- 根据空白拆分单词。\n",
    "- 删除单词中的所有标点符号以减小词汇量（例如，“ What？”变为“ What”）。\n",
    "- 删除所有非字母的单词，以删除独立的标点符号。\n",
    "- 将所有单词标准化为小写以减小词汇量。\n",
    "\n",
    "词汇量对语言建模来说很重要。词汇量越小，模型的训练速度就越快。\n",
    "\n",
    "我们可以按一个函数的顺序执行这些清洁操作。下面是函数clean_doc（）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string\n",
    " \n",
    "# turn a doc into clean tokens\n",
    "def clean_doc(doc):\n",
    "\t# replace '--' with a space ' '\n",
    "\tdoc = doc.replace('--', ' ')\n",
    "\t# split into tokens by white space\n",
    "\ttokens = doc.split()\n",
    "\t# remove punctuation from each token\n",
    "\ttable = str.maketrans('', '', string.punctuation)\n",
    "\ttokens = [w.translate(table) for w in tokens]\n",
    "\t# remove remaining tokens that are not alphabetic\n",
    "\ttokens = [word for word in tokens if word.isalpha()]\n",
    "\t# make lower case\n",
    "\ttokens = [word.lower() for word in tokens]\n",
    "\treturn tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['book', 'i', 'i', 'went', 'down', 'yesterday', 'to', 'the', 'piraeus', 'with', 'glaucon', 'the', 'son', 'of', 'ariston', 'that', 'i', 'might', 'offer', 'up', 'my', 'prayers', 'to', 'the', 'goddess', 'bendis', 'the', 'thracian', 'artemis', 'and', 'also', 'because', 'i', 'wanted', 'to', 'see', 'in', 'what', 'manner', 'they', 'would', 'celebrate', 'the', 'festival', 'which', 'was', 'a', 'new', 'thing', 'i', 'was', 'delighted', 'with', 'the', 'procession', 'of', 'the', 'inhabitants', 'but', 'that', 'of', 'the', 'thracians', 'was', 'equally', 'if', 'not', 'more', 'beautiful', 'when', 'we', 'had', 'finished', 'our', 'prayers', 'and', 'viewed', 'the', 'spectacle', 'we', 'turned', 'in', 'the', 'direction', 'of', 'the', 'city', 'and', 'at', 'that', 'instant', 'polemarchus', 'the', 'son', 'of', 'cephalus', 'chanced', 'to', 'catch', 'sight', 'of', 'us', 'from', 'a', 'distance', 'as', 'we', 'were', 'starting', 'on', 'our', 'way', 'home', 'and', 'told', 'his', 'servant', 'to', 'run', 'and', 'bid', 'us', 'wait', 'for', 'him', 'the', 'servant', 'took', 'hold', 'of', 'me', 'by', 'the', 'cloak', 'behind', 'and', 'said', 'polemarchus', 'desires', 'you', 'to', 'wait', 'i', 'turned', 'round', 'and', 'asked', 'him', 'where', 'his', 'master', 'was', 'there', 'he', 'is', 'said', 'the', 'youth', 'coming', 'after', 'you', 'if', 'you', 'will', 'only', 'wait', 'certainly', 'we', 'will', 'said', 'glaucon', 'and', 'in', 'a', 'few', 'minutes', 'polemarchus', 'appeared', 'and', 'with', 'him', 'adeimantus', 'glaucons', 'brother', 'niceratus', 'the', 'son', 'of', 'nicias', 'and', 'several', 'others', 'who', 'had', 'been', 'at', 'the', 'procession', 'polemarchus', 'said']\n",
      "Total Tokens: 118684\n",
      "Unique Tokens: 7409\n"
     ]
    }
   ],
   "source": [
    "# clean document\n",
    "tokens = clean_doc(doc)\n",
    "print(tokens[:200])\n",
    "print('Total Tokens: %d' % len(tokens))\n",
    "print('Unique Tokens: %d' % len(set(tokens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 保存纯文本\n",
    "我们可以将长长的令牌列表组织成50个输入字和1个输出字的序列。\n",
    "\n",
    "即51个单词的序列。\n",
    "\n",
    "为此，我们可以迭代从令牌51开始的令牌列表，并将之前的50个令牌作为序列，然后重复此过程直到令牌列表的末尾。\n",
    "\n",
    "我们会将令牌转换成以空格分隔的字符串，以供以后存储在文件中。\n",
    "\n",
    "下面列出了将干净令牌列表分成长度为51个令牌的序列的代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Sequences: 118633\n"
     ]
    }
   ],
   "source": [
    "# organize into sequences of tokens\n",
    "length = 50 + 1\n",
    "sequences = list()\n",
    "for i in range(length, len(tokens)):\n",
    "\t# select sequence of tokens\n",
    "\tseq = tokens[i-length:i]\n",
    "\t# convert into a line\n",
    "\tline = ' '.join(seq)\n",
    "\t# store\n",
    "\tsequences.append(line)\n",
    "print('Total Sequences: %d' % len(sequences))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save tokens to file, one dialog per line\n",
    "def save_doc(lines, filename):\n",
    "\tdata = '\\n'.join(lines)\n",
    "\tfile = open(filename, 'w')\n",
    "\tfile.write(data)\n",
    "\tfile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save sequences to file\n",
    "out_filename = 'republic_sequences.txt'\n",
    "save_doc(sequences, out_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用您的文本编辑器查看文件。\n",
    "\n",
    "您会看到每一行都沿一个单词移动，最后有一个新单词可以预测"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 语言模型\n",
    "现在，我们可以从准备的数据中训练统计语言模型。\n",
    "\n",
    "我们将训练的模型是神经语言模型。它具有一些独特的特征：\n",
    "\n",
    "它使用单词的分布式表示，以便具有相似含义的不同单词将具有相似的表示。\n",
    "它在学习模型的同时学习表示。\n",
    "它学习使用最近100个单词的上下文来预测下一个单词的概率。\n",
    "具体来说，我们将使用嵌入层来学习单词的表示，并使用长短期记忆（LSTM）递归神经网络来学习根据上下文来预测单词。\n",
    "\n",
    "让我们从加载训练数据开始。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 加载顺序\n",
    "我们可以使用上一节中开发的load_doc（）函数加载训练数据。\n",
    "\n",
    "加载后，我们可以根据新行进行拆分，将数据拆分为单独的训练序列。\n",
    "\n",
    "下面的代码片段将从当前工作目录中加载“ republic_sequences.txt ”数据文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load doc into memory\n",
    "def load_doc(filename):\n",
    "\t# open the file as read only\n",
    "\tfile = open(filename, 'r')\n",
    "\t# read all text\n",
    "\ttext = file.read()\n",
    "\t# close the file\n",
    "\tfile.close()\n",
    "\treturn text\n",
    " \n",
    "# load\n",
    "in_filename = 'republic_sequences.txt'\n",
    "doc = load_doc(in_filename)\n",
    "lines = doc.split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'book i i went down yesterday to the piraeus with glaucon the son of ariston that i might offer up my prayers to the goddess bendis the thracian artemis and also because i wanted to see in what manner they would celebrate the festival which was a new thing i was'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 编码序列\n",
    "单词嵌入层期望输入序列由整数组成。\n",
    "\n",
    "我们可以将词汇表中的每个单词映射到唯一的整数并编码输入序列。稍后，当我们进行预测时，我们可以将预测转换为数字，并在同一映射中查找它们的关联单词。\n",
    "\n",
    "为了进行这种编码，我们将使用Keras API中的Tokenizer类。\n",
    "\n",
    "首先，必须在整个训练数据集上训练Tokenizer，这意味着它会找到数据中所有唯一的单词，并为每个单词分配一个唯一的整数。\n",
    "\n",
    "然后，我们可以使用fit Tokenizer对所有训练序列进行编码，将每个序列从单词列表转换为整数列表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/likaiwei/anaconda2/envs/ipykernel_py3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "# integer encode sequences of words\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(lines)\n",
    "sequences = tokenizer.texts_to_sequences(lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1046,\n",
       " 11,\n",
       " 11,\n",
       " 1045,\n",
       " 329,\n",
       " 7409,\n",
       " 4,\n",
       " 1,\n",
       " 2873,\n",
       " 35,\n",
       " 213,\n",
       " 1,\n",
       " 261,\n",
       " 3,\n",
       " 2251,\n",
       " 9,\n",
       " 11,\n",
       " 179,\n",
       " 817,\n",
       " 123,\n",
       " 92,\n",
       " 2872,\n",
       " 4,\n",
       " 1,\n",
       " 2249,\n",
       " 7408,\n",
       " 1,\n",
       " 7407,\n",
       " 7406,\n",
       " 2,\n",
       " 75,\n",
       " 120,\n",
       " 11,\n",
       " 1266,\n",
       " 4,\n",
       " 110,\n",
       " 6,\n",
       " 30,\n",
       " 168,\n",
       " 16,\n",
       " 49,\n",
       " 7405,\n",
       " 1,\n",
       " 1609,\n",
       " 13,\n",
       " 57,\n",
       " 8,\n",
       " 549,\n",
       " 151,\n",
       " 11,\n",
       " 57]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以将单词映射为整数，作为Tokenizer对象上名为word_index的字典属性。\n",
    "\n",
    "我们需要知道词汇表的大小，以便以后定义嵌入层。我们可以通过计算映射字典的大小来确定词汇量。\n",
    "\n",
    "为单词分配的值从1到单词总数（例如7,409）。嵌入层需要为此词汇表中的每个单词从索引1到最大索引分配一个矢量表示，并且由于数组的索引是零偏移量，因此单词末尾的单词索引将为7,409；这意味着数组的长度必须为7,409 + 1。\n",
    "\n",
    "因此，当为“嵌入”层指定词汇量时，我们将其指定为比实际词汇量大1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vocabulary size\n",
    "vocab_size = len(tokenizer.word_index) + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 序列输入和输出\n",
    "\n",
    "现在我们已经对输入序列进行了编码，我们需要将它们分为输入（X）和输出（y）元素。\n",
    "\n",
    "我们可以通过数组切片来做到这一点。\n",
    "\n",
    "分离后，我们需要对输出字进行热编码。这意味着将其从整数转换为向量，词汇表中每个单词对应一个值，其中1表示在单词整数值的索引处的特定单词。\n",
    "\n",
    "这样一来，模型就可以预测下一个单词的概率分布。\n",
    "\n",
    "Keras提供了to_categorical（），可用于对每个输入-输出序列对的输出字进行热编码。\n",
    "\n",
    "最后，我们需要向Embedding层指定输入序列的长度。我们知道有50个词是因为我们设计了模型，但是一种好的通用方法是使用输入数据形状的第二维（列数）。这样，如果在准备数据时更改了序列的长度，则无需更改此数据加载代码。它是通用的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from numpy import array\n",
    "from keras.utils import to_categorical\n",
    "\n",
    "# separate into input and output\n",
    "sequences = array(sequences)\n",
    "X, y = sequences[:,:-1], sequences[:,-1]\n",
    "y = to_categorical(y, num_classes=vocab_size)\n",
    "seq_length = X.shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 拟合模型\n",
    "现在，我们可以在训练数据上定义和拟合我们的语言模型。\n",
    "\n",
    "学习的嵌入需要知道词汇量和输入序列的长度，如前所述。它还具有一个参数，用于指定将使用多少维来表示每个单词。即，嵌入向量空间的大小。\n",
    "\n",
    "常见值为50、100和300。在这里我们将使用50，但考虑测试较小或较大的值。\n",
    "\n",
    "我们将使用两个LSTM隐藏层，每个包含100个存储单元。更多的存储单元和更深的网络可能会获得更好的结果。\n",
    "\n",
    "具有100个神经元的密集的完全连接层连接到LSTM隐藏层，以解释从序列中提取的特征。输出层将下一个单词作为词汇量大小的单个矢量预测，并为词汇中的每个单词提供概率。softmax激活函数用于确保输出具有归一化概率的特征。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 50, 50)            370500    \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 50, 100)           60400     \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 100)               80400     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 100)               10100     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 7410)              748410    \n",
      "=================================================================\n",
      "Total params: 1,269,810\n",
      "Trainable params: 1,269,810\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Embedding\n",
    "\n",
    "# define model\n",
    "model = Sequential()\n",
    "model.add(Embedding(vocab_size, 50, input_length=seq_length))\n",
    "model.add(LSTM(100, return_sequences=True))\n",
    "model.add(LSTM(100))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(vocab_size, activation='softmax'))\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "118633/118633 [==============================] - 597s 5ms/step - loss: 5.7583 - acc: 0.1037\n",
      "Epoch 2/10\n",
      "118633/118633 [==============================] - 552s 5ms/step - loss: 5.4834 - acc: 0.1331\n",
      "Epoch 3/10\n",
      "118633/118633 [==============================] - 480s 4ms/step - loss: 5.2847 - acc: 0.1480\n",
      "Epoch 4/10\n",
      "118633/118633 [==============================] - 494s 4ms/step - loss: 5.1461 - acc: 0.1594\n",
      "Epoch 5/10\n",
      "118633/118633 [==============================] - 489s 4ms/step - loss: 5.0340 - acc: 0.1667\n",
      "Epoch 6/10\n",
      "118633/118633 [==============================] - 487s 4ms/step - loss: 4.9369 - acc: 0.1727\n",
      "Epoch 7/10\n",
      "118633/118633 [==============================] - 7101s 60ms/step - loss: 4.8462 - acc: 0.1780\n",
      "Epoch 8/10\n",
      "118633/118633 [==============================] - 487s 4ms/step - loss: 4.7584 - acc: 0.1832\n",
      "Epoch 9/10\n",
      "118633/118633 [==============================] - 481s 4ms/step - loss: 4.6733 - acc: 0.1877\n",
      "Epoch 10/10\n",
      "118633/118633 [==============================] - 528s 4ms/step - loss: 4.5891 - acc: 0.1923\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x10ce854a8>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compile model\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "# fit model\n",
    "model.fit(X, y, batch_size=128, epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 保存模型\n",
    "运行结束时，经过训练的模型将保存到file中。\n",
    "\n",
    "在这里，我们使用Keras模型API将模型保存到当前工作目录中的文件“ model.h5 ”。\n",
    "\n",
    "稍后，当我们加载模型进行预测时，我们还将需要将单词映射为整数。这在Tokenizer对象中，我们也可以使用Pickle保存它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pickle import dump\n",
    "\n",
    "# save the model to file\n",
    "model.save('model.h5')\n",
    "# save the tokenizer\n",
    "dump(tokenizer, open('tokenizer.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 完整的例子\n",
    "我们可以将所有这些放在一起。下面列出了适合语言模型的完整示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 50, 50)            370500    \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 50, 100)           60400     \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 100)               80400     \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 100)               10100     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 7410)              748410    \n",
      "=================================================================\n",
      "Total params: 1,269,810\n",
      "Trainable params: 1,269,810\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/100\n",
      "118633/118633 [==============================] - 605s 5ms/step - loss: 6.1962 - acc: 0.0667\n",
      "Epoch 2/100\n",
      "118633/118633 [==============================] - 618s 5ms/step - loss: 5.7788 - acc: 0.0985\n",
      "Epoch 3/100\n",
      "118633/118633 [==============================] - 818s 7ms/step - loss: 5.5498 - acc: 0.1220\n",
      "Epoch 4/100\n",
      " 19456/118633 [===>..........................] - ETA: 8:30 - loss: 5.3755 - acc: 0.1375"
     ]
    }
   ],
   "source": [
    "from numpy import array\n",
    "from pickle import dump\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.utils import to_categorical\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM\n",
    "from keras.layers import Embedding\n",
    "\n",
    "# load doc into memory\n",
    "def load_doc(filename):\n",
    "\t# open the file as read only\n",
    "\tfile = open(filename, 'r')\n",
    "\t# read all text\n",
    "\ttext = file.read()\n",
    "\t# close the file\n",
    "\tfile.close()\n",
    "\treturn text\n",
    "\n",
    "# load\n",
    "in_filename = 'republic_sequences.txt'\n",
    "doc = load_doc(in_filename)\n",
    "lines = doc.split('\\n')\n",
    "\n",
    "# integer encode sequences of words\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(lines)\n",
    "sequences = tokenizer.texts_to_sequences(lines)\n",
    "# vocabulary size\n",
    "vocab_size = len(tokenizer.word_index) + 1\n",
    "\n",
    "# separate into input and output\n",
    "sequences = array(sequences)\n",
    "X, y = sequences[:,:-1], sequences[:,-1]\n",
    "y = to_categorical(y, num_classes=vocab_size)\n",
    "seq_length = X.shape[1]\n",
    "\n",
    "# define model\n",
    "model = Sequential()\n",
    "model.add(Embedding(vocab_size, 50, input_length=seq_length))\n",
    "model.add(LSTM(100, return_sequences=True))\n",
    "model.add(LSTM(100))\n",
    "model.add(Dense(100, activation='relu'))\n",
    "model.add(Dense(vocab_size, activation='softmax'))\n",
    "print(model.summary())\n",
    "# compile model\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "# fit model\n",
    "model.fit(X, y, batch_size=128, epochs=100)\n",
    "\n",
    "# save the model to file\n",
    "model.save('model.h5')\n",
    "# save the tokenizer\n",
    "dump(tokenizer, open('tokenizer.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用语言模型\n",
    "现在我们有了训练有素的语言模型，就可以使用它了。\n",
    "\n",
    "在这种情况下，我们可以使用它来生成与源文本具有相同统计属性的新文本序列。\n",
    "\n",
    "我们将从重新加载训练序列开始。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 载入资料\n",
    "我们可以使用上一部分中的相同代码来加载文本的训练数据序列。\n",
    "\n",
    "具体来说，就是load_doc（）函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load doc into memory\n",
    "def load_doc(filename):\n",
    "\t# open the file as read only\n",
    "\tfile = open(filename, 'r')\n",
    "\t# read all text\n",
    "\ttext = file.read()\n",
    "\t# close the file\n",
    "\tfile.close()\n",
    "\treturn text\n",
    " \n",
    "# load cleaned text sequences\n",
    "in_filename = 'republic_sequences.txt'\n",
    "doc = load_doc(in_filename)\n",
    "lines = doc.split('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们需要文本，以便我们可以选择源序列作为模型输入，以生成新的文本序列。\n",
    "\n",
    "该模型将需要100个单词作为输入。\n",
    "\n",
    "稍后，我们将需要指定预期的输入长度。我们可以从输入序列中确定这一点，方法是计算已加载数据的一行的长度，并对同样在同一行的预期输出字减去1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seq_length = len(lines[0].split()) - 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 加载模型\n",
    "现在，我们可以从文件中加载模型。\n",
    "\n",
    "Keras提供了load_model（）函数来加载模型，以供使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the model\n",
    "model = load_model('model.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the tokenizer\n",
    "tokenizer = load(open('tokenizer.pkl', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 产生文字\n",
    "生成文本的第一步是准备种子输入。\n",
    "\n",
    "为此，我们将从输入文本中选择随机文本行。选择之后，我们将打印它，以便我们对使用的内容有所了解。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select a seed text\n",
    "seed_text = lines[randint(0,len(lines))]\n",
    "print(seed_text + '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，我们可以一次生成一个新单词。\n",
    "\n",
    "首先，必须使用与训练模型时相同的标记器将种子文本编码为整数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoded = tokenizer.texts_to_sequences([seed_text])[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "该模型可以通过调用model.predict_classes（）直接预测下一个单词，该函数将以最高的概率返回该单词的索引。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict probabilities for each word\n",
    "yhat = model.predict_classes(encoded, verbose=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们可以在Tokenizers映射中查找索引以获取关联的单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_word = ''\n",
    "for word, index in tokenizer.word_index.items():\n",
    "\tif index == yhat:\n",
    "\t\tout_word = word\n",
    "\t\tbreak"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们可以将此单词附加到种子文本中并重复该过程。\n",
    "\n",
    "重要的是，输入序列将变得太长。在将输入序列编码为整数之后，我们可以将其截断为所需的长度。Keras提供了pad_sequences（）函数，可用于执行此截断。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以将所有这些包装到一个称为generate_seq（）的函数中，该函数将模型，标记器，输入序列长度，种子文本和要生成的单词数作为输入。然后，它返回由模型生成的单词序列。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate a sequence from a language model\n",
    "def generate_seq(model, tokenizer, seq_length, seed_text, n_words):\n",
    "\tresult = list()\n",
    "\tin_text = seed_text\n",
    "\t# generate a fixed number of words\n",
    "\tfor _ in range(n_words):\n",
    "\t\t# encode the text as integer\n",
    "\t\tencoded = tokenizer.texts_to_sequences([in_text])[0]\n",
    "\t\t# truncate sequences to a fixed length\n",
    "\t\tencoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n",
    "\t\t# predict probabilities for each word\n",
    "\t\tyhat = model.predict_classes(encoded, verbose=0)\n",
    "\t\t# map predicted word index to word\n",
    "\t\tout_word = ''\n",
    "\t\tfor word, index in tokenizer.word_index.items():\n",
    "\t\t\tif index == yhat:\n",
    "\t\t\t\tout_word = word\n",
    "\t\t\t\tbreak\n",
    "\t\t# append to input\n",
    "\t\tin_text += ' ' + out_word\n",
    "\t\tresult.append(out_word)\n",
    "\treturn ' '.join(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "放在一起，下面列出了用于从学习语言模型生成文本的完整代码清单。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "better be referred to damon himself for the analysis of the subject would be difficult you know socrates expresses himself carelessly in accordance with his assumed ignorance of the details of the subject in the first part of the sentence he appears to be speaking of paeonic rhythms which are in\n",
      "\n",
      "the state and the same and the same and the same and the other and the other and the other and the other and the other and the other and the other and the other and the other and the other and the other and the other and the other\n"
     ]
    }
   ],
   "source": [
    "from random import randint\n",
    "from pickle import load\n",
    "from keras.models import load_model\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    " \n",
    "# load doc into memory\n",
    "def load_doc(filename):\n",
    "\t# open the file as read only\n",
    "\tfile = open(filename, 'r')\n",
    "\t# read all text\n",
    "\ttext = file.read()\n",
    "\t# close the file\n",
    "\tfile.close()\n",
    "\treturn text\n",
    " \n",
    "# generate a sequence from a language model\n",
    "def generate_seq(model, tokenizer, seq_length, seed_text, n_words):\n",
    "\tresult = list()\n",
    "\tin_text = seed_text\n",
    "\t# generate a fixed number of words\n",
    "\tfor _ in range(n_words):\n",
    "\t\t# encode the text as integer\n",
    "\t\tencoded = tokenizer.texts_to_sequences([in_text])[0]\n",
    "\t\t# truncate sequences to a fixed length\n",
    "\t\tencoded = pad_sequences([encoded], maxlen=seq_length, truncating='pre')\n",
    "\t\t# predict probabilities for each word\n",
    "\t\tyhat = model.predict_classes(encoded, verbose=0)\n",
    "\t\t# map predicted word index to word\n",
    "\t\tout_word = ''\n",
    "\t\tfor word, index in tokenizer.word_index.items():\n",
    "\t\t\tif index == yhat:\n",
    "\t\t\t\tout_word = word\n",
    "\t\t\t\tbreak\n",
    "\t\t# append to input\n",
    "\t\tin_text += ' ' + out_word\n",
    "\t\tresult.append(out_word)\n",
    "\treturn ' '.join(result)\n",
    " \n",
    "# load cleaned text sequences\n",
    "in_filename = 'republic_sequences.txt'\n",
    "doc = load_doc(in_filename)\n",
    "lines = doc.split('\\n')\n",
    "seq_length = len(lines[0].split()) - 1\n",
    " \n",
    "# load the model\n",
    "model = load_model('model.h5')\n",
    " \n",
    "# load the tokenizer\n",
    "tokenizer = load(open('tokenizer.pkl', 'rb'))\n",
    " \n",
    "# select a seed text\n",
    "seed_text = lines[randint(0,len(lines))]\n",
    "print(seed_text + '\\n')\n",
    " \n",
    "# generate new text\n",
    "generated = generate_seq(model, tokenizer, seq_length, seed_text, 50)\n",
    "print(generated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 扩展\n",
    "本节列出了一些扩展的想法。\n",
    "\n",
    "- 句。根据句子拆分原始数据，并将每个句子填充到固定长度（例如最长的句子长度）。\n",
    "- 简化词汇。探索更简单的词汇，去除词干或停用词。\n",
    "- 音调模型。调整模型，例如嵌入的大小或隐藏层中的存储单元数，以查看是否可以开发更好的模型。\n",
    "- 更深层次的模型。将模型扩展为具有多个LSTM隐藏层，也许可以通过Dropout来查看是否可以开发更好的模型。\n",
    "- 预训练词嵌入。扩展模型以使用预训练的word2vec或GloVe向量，以查看是否可以得到更好的模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
